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PEO-IWS  ACB  Insertion  Portfolio  Optimization! 

Johnathan  Mun,  Tom  Housel  and  Mark  D.  Wessman 


Overview 

Program  Executive  Office-Integrated  Warfare  Systems  (PEO-IWS)  engaged  a  team 
from  the  Naval  Postgraduate  School  (NPS)  to  conduct  a  pilot  study  to  apply  the  Knowledge 
Valued  Added  +  Real  Options  +  Integrated  Risk  Management  +  Portfolio  Optimization  (KVA 
+  RO  +  IRM  +  PO)  method  to  estimate  the  value  stream  created  by  the  capabilities  to  be 
inserted  within  the  Aegis  Weapons  System  (AWS)  through  the  Advanced  Capability  Build 
(ACB)  process — as  described  in  the  PEO-IWS  Surface  Combat  System  Acquisition 
Management  Plan  (AMP) — given  budget  constraints  and  ship  industrial  availability 
schedules.  The  goal  was  to  determine  what  order  of  capability  insertion  provided  the  best 
returns  within  an  optimized  portfolio,  treating  each  capability  as  a  real  option.  The  KVA  +  RO 
+  IRM  +  PO  approach  was  used  to  estimate  the  warfighter  value  delivered  by  each 
capability  within  the  context  of  a  portfolio-optimization,  integrated  risk-management  model. 
The  results  provide  a  set  of  options  based  on  selected  constraints  for  insertion  of  the 
capabilities  over  the  period  of  interest  (Fiscal  Years  2014-2025)  based  on  an  optimized 
portfolio  model.  For  detailed  information  on  the  KVA  +  RO  +IRM  +  PO  approach,  see  the 
technical  appendix  to  this  report. 

The  pilot  study  analysis  articulated  a  notional  value  measure  of  military  capability  for 
a  specified  set  of  23  capabilities  to  be  considered,  and  examined  four  discrete  approaches: 
(1)  a  ranking  by  value  within  a  constrained  total  integration  budget  (optimal  on  budget),  (2)  a 
similar  ranking  by  value  and  budget  with  a  risk  constraint  added  (optimal  cost-risk),  (3) 
ranking  by  value  constrained  by  integration  budget,  with  the  additional  constraint  that  a 
particular  capability  must  be  included  in  the  first  increment  (Capability  2  must-have),  and  (4) 
the  portfolio  with  the  specified  capability  in  the  first  increment  and  a  risk  constraint  added 
(Capability  2  cost-risk).  Each  approach  generated  a  distinct  recommendation  for  the 
composition  and  sequencing  of  the  capabilities  within  the  ACB  schedule.  Under  the  ACB 
model,  capabilities  will  be  inserted  within  the  AWS  system  every  two  years.  ACBs  are 
identified  by  the  fiscal  year  in  which  the  first  ship  receives  the  software  upgrade.  The 
analysis  encompassed  ACB  14  (2014)  through  ACB  18  (2018).  The  analysis  included  all  the 
ships  that  would  receive  the  ACBs  for  each  two-year  period  from  2014-2025,  and 
accounted  for  the  ships  being  phased  into  the  program  through  scheduled  repair  availability 
periods.  This  period  was  selected  because  at  the  end  of  that  time,  all  ships  with  the  AWS 
would  have  been  inducted  into  the  process.  The  analysis  assumes  that  value  would  begin  to 
accrue  for  a  given  ship  as  soon  as  an  ACB  was  implemented  in  that  ship  and  would 
continue  to  accrue  through  subsequent  ACBs  throughout  the  service  life  of  the  ship.  ACBs 
beyond  ACB  18  were  not  considered  for  the  pilot  analysis,  but  both  additional  capabilities 
and  future  ACBs  can  easily  be  added  to  the  analysis.  Within  the  ACB  process,  a  ship 


The  analyses  performed  (Monte  Carlo  risk  simulation,  dynamic  optimization,  and  real  options  analysis)  apply  the  Risk  Simulator  software 
and  ROV  Modeler  software  tools  available  from  Real  Options  Valuation,  Inc.  (www.realoptionsvaluation.com),  and  the  software  screen 
shots  were  reprinted  with  their  permission.  Although  there  exist  several  commercial  off-the-shelf  software  products  available  for  running 
optimization,  Risk  Simulator  and  ROV  Modeler  were  the  only  tools  found  to  be  suitable  due  to  their  ability  in  handling  real  options 
analysis,  stochastic  optimization,  risk  simulation,  and  other  requirements  in  the  analyses  performed.  These  software  tools  were  developed 
by  the  author,  Dr.  Johnathan  Mun. 
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receives  the  current  ACB,  as  well  as  the  previous  ACBs  not  yet  completed.  For  example,  a 
ship  entering  the  program  with  ACB  18  also  receives  the  capabilities  from  ACB  14  and  ACB 
16  at  the  same  time.  A  ship  that  receives  the  first  ACB  as  ACB  14  will  receive  its  next 
update  with  ACB  18  but  will  receive  the  ACB  16  capabilities  at  the  same  time.  Aggregation 
of  the  value  for  individual  ships  provides  a  measure  in  terms  of  capability-ship-years. 

Within  the  context  of  the  KVA  approach,  the  study  assumed  that  a  relatively  objective 
and  extensible  metric  for  military  value  was  the  relative  complexity  of  the  software  modules 
that  implement  each  capability  in  the  open  architected  AWS,  and  that  the  complexity  of  the 
component  could  be  represented  by  the  relative  magnitude  of  the  number  of  delivered 
source  lines  of  code  (DSLOC),  given  that  the  programming  languages  used  were 
comparable.  A  second  measure  of  relative  military  value  was  achieved  through  collection  of 
subject-matter  expert  (SME)  rankings  of  relative  component  complexity  and  mission 
criticality.  The  study  methodology  aggregated  the  component  data  to  the  capability  level 
using  SME  mapping  of  components  to  capabilities.  These  measures  correlated  highly  with 
the  DSLOC  rankings,  providing  a  validation  of  the  assumption  that  relative  magnitude  of 
DSLOC  can  provide  a  measure  of  military  value.  SME  estimates  of  the  complexity  of  code 
and  DSLOC  were  presumed  to  be  those  of  the  capability  components  themselves  (versus 
simply  the  integration)  since  warfighting  functionality  (the  military  value  of  the  system)  was 
implemented  by  the  component.  Integrating  the  components/capabilities  into  the  system 
made  the  warfighting  functions  of  the  components/capabilities  available  to  the  user. 

The  cost  basis  for  each  insertion  employed  for  the  study  was  based  on  an 
aggregated  average  of  high-,  medium-,  and  low-cost  estimates  to  integrate  the  required 
components  into  the  AWS  for  given  capability  insertions.  Cost  of  integration  was  used  as  the 
key  cost  parameter  to  provide  the  analysis  based  on  PEO-IWS  1’s  perspective  as  the 
integrating  agent,  versus  as  a  component  provider.  The  correlations  among  the  subjective 
measure  of  military  value  and  relative  complexity  of  components  derived  from  judgments  of 
SMEs  from  PEO-IWS  1.0  and  the  Naval  Surface  Warfare  Center  Dahlgren  Division 
(NSWCDD)  were  very  high,  indicating  that  the  estimates  were  reliable  (Table  1).  The 
correlation  between  costs  for  the  insertions  and  value  described  using  this  method  was  very 
low,  indicating  that  integration  cost  does  not  predict  military  value.  These  findings  support 
the  need  for  using  the  KVA  method  to  determine  expected  military  value  (EMV),  the 
objective  measure  used  in  portfolio  optimization  and  selection. 

The  toolset  applied — using  real  options,  portfolio  optimization,  and  integrated  risk 
management — provides  a  means  for  quickly  estimating  the  effects  of  various  capability 
insertions  over  the  period  of  interest.  It  provides  management  with  the  flexibility  to  examine 
various  ACB  capability  insertion  options  given  budget  and  ship  availability  constraints.  The 
analysis  for  this  study  employed  the  following  steps: 

Data  collection  and  analysis  to  determine  the  best  proxy  for  Expected  Military 
Value  (EMV),  using  objective  data  on  DSLOC,  subject-matter  estimates  of 
complexity  and  mission  criticality  for  each  capability,  and  OPNAV  (Chief  of 
Naval  Operations  staff)  sponsor  and  technical  community  priorities  for  each 
capability. 

Static  and  dynamic  optimization  runs  based  on  four  different  EMV  measures 
for  multi-criteria  optimization,  to  determine  the  best  allocation  and  selection  of 
capabilities  given  a  nominal  $150  million  budget  constraint  for  each  ACB, 
using  a  range  of  cost  estimates  for  integration  provided  by  SMEs. 
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3.  Combination  of  the  four  EMV  methods  to  obtain  the  portfolio  of  options, 
results  and  recommendations  for  sequencing  capability  insertions  for  ACB 
14,  ACB  16,  and  ACB  18. 

4.  Computation  of  aggregate  EMV  values  through  the  insertions  of  ACBs  using 
actual  planned  ship-availability  schedules  as  published  in  the  Surface  Ship 
Acquisition  Management  Plan  (AMP). 

5.  Monte  Carlo  risk  simulation  of  cost  estimates  to  determine  risk  of  budget  and 
cost  overruns. 

6.  Generation  of  an  alternate  scenario  by  applying  OPNAV’s  Priority  1  capability 
(Capability  2)  as  a  “must-have”  in  the  portfolio  selection,  and  identification  of 
what  other  capabilities  should  be  inserted  in  such  a  scenario  and  how  this 
would  affect  accrued  value  over  the  ACB  insertion  timeframe. 

7.  Determination  of  a  Portfolio  Efficient  Frontier,  in  which  we  determine  multiple 
scenarios  of  increasing  budget  (i.e.,  if  the  $150  million  budget  were  increased 
to  $200  million,  or  $250  million,  or  $300  million,  etc.,  what  will  the  optimal 
portfolio  look  like  for  each  budget;  what  capabilities  should  be  added  or 
replaced,  and  what  would  be  their  impacts  on  EMV?). 

8.  Repetition  of  the  previous  analyses — with  an  additional  constraint  that  the 
portfolio  selected  must  have  an  85%  or  greater  probability  of  completing 
within  budget. 

As  follow-on  to  the  work  documented  in  this  report,  PEO-IWS  should  consider  the 
following: 

1 .  Strategic  Real  Options  or  analysis  of  alternatives,  examining  various  courses 
of  action  should  certain  capabilities  be  linked  or  nested  with  respect  to 
another  (e.g.,  there  might  be  a  “platform  capability”  that  might  have  a  high 
initial  cost  but  bring  significant  downstream  options  for  add-on  capabilities 
with  significant  EMV.  Or,  there  may  be  mutually  exclusive  or  dependent 
capabilities — with  which  the  implementation  of  capability  precludes  another 
from  being  implemented  or  requires  another  to  be  implemented,  or  will 
reduce  the  cost  and  increase  the  total  EMV  of  another  capability  when  they 
are  implemented  together). 

2.  Additional  modeling,  such  as  adding  new  capabilities  to  the  list,  adding 
considerations  of  additional  risk  factors  (e.g.,  technical,  schedule). 

3.  Training  and  software  implementation  for  risk  simulation  and  optimization. 

Adoption  of  the  foregoing  will  lead  to  a  more  refined  and  robust  analysis  of  the  value, 
risk,  and  cost  of  future  options  for  capability  insertions  for  the  Aegis  system.  The  remainder 
of  the  report  is  sequenced  as  follows: 

•  Statement  of  Work  (SOW)  Objectives 

•  Problem  Formulation 

•  Methodology 

Table  1.  Data  Collection 

Table  2.  Portfolio  Optimization 

ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY  740 
NAVAL  POSTGRADUATE  SCHOOL 


Table  3.  Expected  Military  Value 

Table  4.  Sequencing  of  ACB  14,  ACB  16,  ACB  18 

Table  5.  Alternate  Scenario:  Capability  2  Must-Have 

Table  6.  Aggregate  EMV 

Table  7.  Monte  Carlo  Risk  Simulation 

Table  8.  Portfolio  Optimization’s  Efficient  Frontier 

Table  9.  Optimization  with  a  Risk  Constraint 

Statement  of  Work  Objectives 

The  focus  of  this  work  is  to  conduct  a  pilot  study  to  provide  return  on  investment  and 
real  options/portfolio  optimization  analysis  to  help  articulate  value  proposition  in  selection  of 
capabilities  for  inclusion  in  the  modernization  of  the  Aegis  Weapons  System  (AWS)  in  US 
Navy  cruisers  and  destroyers.  The  analysis  was  to  be  demonstrated  in  a  manner  that  would 
support  the  next  budget  submission  cycle.  In  addition,  the  project  was  to  use  the  Knowledge 
Value  Added  +  Real  Options  +  Integrated  Risk  Management  +  Portfolio  Optimization  (KVA  + 
RO  +  IRM  +  PO)  methodology,  with  supporting  software,  to  aid  in  the  process  performance 
analysis  and  option-value  estimation.  The  customer  selected  the  processes  and  systems  for 
the  analysis  to  establish  the  baseline  return  on  investment  (ROI)  estimates.  This  project 
focused  on  conducting  the  KVA  +  RO  +  IRM  +  PO  analysis  on  the  identified  ACB  insertion 
options  by  working  with  PEO-IWS  1  and  NSWCDD  personnel  to  establish  the  necessary 
baselines  and  analyses  and,  concurrently,  to  lay  the  foundation  for  developing  the  level  of 
knowledge  necessary  for  the  organization  to  use  and  maintain  the  toolset  going  forward. 
This  approach  ensures  that  the  managers  of  the  process  have  a  decision  toolset  and  the 
knowledge  to  interpret  the  results  of  the  analysis  outputs.  These  tools  include  the 
applications  of  risk  analysis,  forecasting,  risk  hedging  and  management  strategies,  strategic 
real-options  applications,  project  portfolio  optimization  and  selection,  and  other  related 
analytics.  In  addition,  aggregate  numbers  used  to  support  the  building  of  a  business  case  to 
meet  the  acquisition  community  requirements  for  the  selected  problem  space  were  also  to 
be  documented.  Management-level  reports  were  provided  to  evaluate  ongoing  OA 
acquisition  initiatives.  The  products  of  the  pilot  were  developed  in  a  manner  that  can  provide 
a  basis  for  extension  and  implementation  across  the  PEO  as  a  method  and  toolset  to  be 
used  on  an  ongoing  basis.  This  extended  use  will  provide  the  ability  to  better  manage 
acquisition  decisions  and  to  make  the  case  for  those  decisions  to  both  sponsors  and  the 
Acquisition  chain  of  command. 

Problem  Formulation 

The  US  Navy  is  constantly  faced  with  many  difficult  portfolio  optimization  decisions. 
These  decisions  include  allocating  financial  resources,  building  or  expanding  facilities  and 
capabilities,  and  determining  acquisition  strategies.  Such  decisions  might  involve  thousands 
or  millions  of  potential  alternatives.  Considering  and  evaluating  each  of  them  would  be 
impractical  or  even  impossible.  A  model  can  provide  valuable  assistance  in  incorporating 
relevant  variables  when  analyzing  decisions  and  finding  the  best  solutions  for  making 
decisions.  Models  capture  the  most  important  features  of  a  problem  and  present  them  in  a 
form  that  is  easy  to  interpret.  Models  often  provide  insights  that  intuition  alone  cannot.  An 
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optimization  model  has  three  major  elements:  decision  variables,  constraints,  and 
objectives.  The  optimization  methodology  finds  the  best  combination  or  permutation  of 
decision  variables  (e.g.,  which  strategies  to  pursue  and  which  projects  to  execute)  in  every 
conceivable  way  such  that  the  objective  is  maximized  (e.g.,  return  on  investment,  military 
value-added,  proxies  for  revenues  and  income)  or  minimized  (e.g.,  risk  and  costs)  while  still 
satisfying  the  constraints  (e.g.,  budget  and  resources). 

In  order  to  obtain  optimal  values,  one  generally  must  search  in  an  iterative  or  ad  hoc 
fashion.  This  search  involves  running  one  iteration  for  an  initial  set  of  values,  analyzing  the 
results,  changing  one  or  more  values,  rerunning  the  model,  and  repeating  the  process  until 
finding  a  satisfactory  solution.  This  process  can  be  very  tedious  and  time-consuming,  even 
for  small  models,  and  often  it  is  not  clear  how  to  adjust  the  values  from  one  iteration  to  the 
next.  A  more  rigorous  method  systematically  enumerates  all  possible  alternatives.  This 
approach  guarantees  optimal  solutions  if  the  model  is  correctly  specified.  If  an  optimization 
model  depends  on  only  two  decision  variables,  and  if  each  variable  has  10  possible  values, 
then  trying  each  combination  requires  100  iterations  (102  alternatives).  If  each  iteration  is 
very  short  (e.g.,  two  seconds),  then  the  entire  process  could  be  done  in  approximately  three 
minutes  of  computer  time.  However,  instead  of  two  decision  variables,  if  the  option  set 
includes  “go”  or  “no-go”  decisions  on  23  alternative  selections — as  in  the  case  of  the  current 
ACB  analysis — then  trying  all  combinations  requires  2.58  x  1022  iterations  of  alternatives.  It 
is  easily  possible  for  complete  enumeration  to  take  months  or  even  years  to  carry  out  on  a 
supercomputer.  Practicality,  then,  demands  that  the  analyst  employ  some  advanced 
algorithms  and  techniques  in  Risk  Simulator  and  ROV  Modeler  for  running  the  portfolio 
selection  and  optimization.  Before  embarking  on  solving  an  optimization  problem,  it  is  vital  to 
understand  the  terminology  of  optimization — the  terms  used  to  describe  certain  attributes  of 
the  optimization  process.  These  words  include  decision  variables,  constraints,  and 
objectives. 

Decision  Variables  are  quantities  over  which  the  decision-makers  have  control;  for 
example,  the  amount  of  a  product  to  make,  the  number  of  dollars  to  allocate  among  different 
investments,  or  which  projects  to  select  from  among  a  limited  set.  As  an  example,  portfolio 
optimization  analysis  includes  a  “go”  or  “no-go”  decision  on  particular  projects.  In  addition, 
the  dollar  or  percentage  budget  allocation  across  multiple  projects  also  can  be  structured  as 
decision  variables. 

Constraints  describe  relationships  among  decision  variables  that  restrict  the  values 
of  the  decision  variables.  For  example,  a  constraint  might  ensure  that  the  total  amount  of 
money  allocated  among  various  investments  cannot  exceed  a  specified  amount  or,  at  most, 
one  project  from  a  certain  group  can  be  selected.  Constraints  also  include  budget  and  timing 
restrictions,  minimum  returns,  or  risk-tolerance  levels. 

Objectives  give  a  mathematical  representation  of  the  model’s  desired  outcome — 
such  as  maximizing  EMV,  benefits,  and  profit,  or  minimizing  cost  and  risk — in  terms  of  the 
decision  variables.  In  financial  analysis,  for  example,  the  objective  may  be  to  maximize 
returns  while  minimizing  risks  (maximizing  the  Sharpe’s  ratio  or  returns-to-risk  ratio). 

The  solution  to  an  optimization  model  provides  a  set  of  values  for  the  decision 
variables  that  optimizes  (maximizes  or  minimizes)  the  associated  objective.  If  the  real 
business  conditions  were  simple,  and  if  the  future  were  predictable,  then  all  data  in  an 
optimization  model  would  be  constant,  making  the  model  deterministic.  In  many  cases, 
however,  a  deterministic  optimization  model  cannot  capture  all  the  relevant  intricacies  of  a 
practical  decision-making  environment.  When  a  model’s  data  are  uncertain  and  can  only  be 
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described  probabilistically,  the  objective  will  have  some  probability  distribution  for  any 
chosen  set  of  decision  variables.  The  analyst  can  find  this  probability  distribution  by 
simulating  the  model  using  Risk  Simulator.  An  optimization  model  under  uncertainty  has 
several  additional  elements,  including  assumptions  and  forecasts. 

Assumptions  capture  the  uncertainty  of  model  data  using  probability  distributions, 
whereas  forecasts  are  the  frequency  distributions  of  possible  results  for  the  model.  Forecast 
statistics  are  summary  values  of  a  forecast  distribution,  such  as  the  mean,  standard 
deviation,  and  variance.  The  optimization  process  controls  the  optimization  by  maximizing  or 
minimizing  the  objective.  Each  optimization  model  has  one  objective,  a  variable  that 
mathematically  represents  the  model’s  objective  in  terms  of  the  assumption  and  decision 
variables.  Optimization’s  job  is  to  find  the  optimal  (minimum  or  maximum)  value  of  the 
objective  by  selecting  and  improving  different  values  for  the  decision  variables.  When  model 
data  are  uncertain  and  can  only  be  described  using  probability  distributions,  the  objective 
itself  will  have  some  probability  distribution  for  any  set  of  decision  variables.  In  the  current 
project’s  optimization  analysis,  the  problem  formulation  is  to  optimize  the  Aegis  ACB 
composition  based  on: 

•  Potential  return  on  capability  (return  on  investment,  expected  military  value, 
and  other  multiple  criteria), 

•  Investment  constraints  (e.g.,  $150  million  per  ACB  cycle), 

•  Ship  schedule  and  availability,  and 

•  Selecting  the  best  combinations  and  permutations  of  capabilities  using  the 
portfolio  optimization  approach  as  a  series  of  options. 

Methodology 

In  this  section,  we  discuss  the  methodology  employed  in  more  detail,  with  particular 
emphasis  on  the  high-level  understanding  of  the  approach  and  the  results.  For  the  technical 
mathematical  constructs,  please  refer  to  the  Appendix  for  additional  technical  background 
readings.  Briefly,  the  methodology  employed  is  divided  into  several  steps,  as  covered  in  the 
following  subsections. 

Data  Collection 

Data  collection  and  analysis  is  the  first  step  employed  to  determine  the  best  proxy  for 
Expected  Military  Value  (EMV)  and  cost  estimates  of  each  capability.  To  that  end,  we  relied 
on  data  ranging  from  objective  values  such  as  delivered  source  lines  of  code  (DSLOC)  of 
software,  semi-objective  measures  such  as  estimates  of  integration  cost  for  each  capability 
(using  high,  most-likely,  and  low  estimates  for  cost,  so  that  we  can  perform  a  Monte  Carlo 
risk  simulation  later),  to  more  subjective  estimates  from  subject-matter  experts  (SMEs)  on 
the  amount  of  functional  complexity  and  operational  criticality  for  each  component.  PEO- 
IWS  representatives  also  provided  OPNAV  and  acquisition  community  priorities  for  each 
capability.  The  analysis  demonstrated  that  complexity  is  proportionate  to  value,  but  there 
were  low  correlations  between  EMV  and  cost  estimates — indicating  that  we  cannot  reliably 
use  cost  alone  as  an  estimate  to  determine  the  best  portfolio  allocation  for  maximizing  EMV. 
The  correlation  matrix  is  shown  in  Tablel. 
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Table  1.  Correlation  Matrix 


Linear  Correlation  Matrix, 

Cost 

SME  Mean 

DSLOC 

Technical 

OPNAV 

EMV:  SMEVA 

EMV:  All 
Weighted  & 
Common  Sized 

EMV:  OPNAV 

EMV:  OPNAV 

Midpoint 

Value  Added 

Complexity 

Priority  H-L 

Priority  H-L 

and  DSLOC 

&  Technical 

&  DSLOC 

Cost  Midpoint 

1 

SME  Mean  Value  Added 

0.07 

1 

DSLOC  Complexity 

0.40 

0.64 

1 

Technical  Priority  H-L 

0.46 

0.42 

0.42 

1 

OPNAV  Priority  H-L 

0.34 

0.40 

0.43 

0.60 

1 

EMV:  SMEVA  and  DSLOC 

0.30 

0.66 

1.00 

0.42 

0.44 

1 

EMV:  All,  Weight,  Sized 

0.36 

0.01 

0.03 

0.65 

0.60 

0.04 

1 

EMV:  OPNAV  &  Technical 

0.41 

0.47 

0.44 

0.07 

0.07 

0.44 

0.69 

1 

EMV:  OPNAV  &  DSLOC 

0.62 

0.00 

0.76 

0.00 

0.06 

0.77 

1 

Risk  and  uncertainty  can  also  be  estimated  based  on  various  criteria  (in  the  current 
analysis,  we  use  cost  uncertainty  as  a  proxy  for  risk),  and  value  is  assumed  to  be  generated 
as  capabilities  are  realized  through  installation  in  specific  ships  over  time.  To  get  started 
with  the  data  collection,  we  had  to  perform  the  following  steps: 

•  Establish  operational  definitions  of  value  and  cost  of  each  ACB  capability 
insertion. 

•  Identify  the  projected  ship  schedule  to  establish  availability  for  ACB  insertions 
every  two  years. 

•  Obtain  SME  identification  and  description  of  ACB  components  and 
capabilities. 

•  Undertake  model  generation  and  iterations  with  various  inputs,  including 
running  cross-correlations  to  determine  the  impact  and  validity  of  SME 
estimates. 

The  data  sources  used  include: 

•  AMP  Ver.  5.4  (27  Oct  2008)  documentation  of  moving  to  an  OA  approach  in 
ACB  insertions, 

•  Ship  schedule,  capability  candidates  for  integration,  components  of  the 
system  to  be  changed  (mapped  to  the  capabilities),  and  integration  cost  for 
each  capability  provided  by  IWS  1 , 

•  SMEs  estimates  of  complexity  and  mission  criticality  for  components,  and 

•  DSLOC  for  each  component. 

Figure  1  shows  a  sample  of  the  collected  data  used  in  the  analysis.  The  analysis 
began  by  assuming  23  capabilities  (more  can  be  added  later  as  required).  Next,  the  analysis 
applied  the  average  of  the  SME  value-added  estimates;  the  high,  most-likely,  and  low  cost 
estimates;  the  OPNAV  and  technical  priorities;  and  the  DSLOC  for  each  component,  as 
specified  in  the  Surface  Combat  System  Objective  Architecture.  Using  these  raw  variables, 
we  generated  various  EMV  metrics  by  accounting  for  the  SME  mean  value-added  estimates 
and  DSLOC,  and  weighting  them — as  well  as  common-sizing  their  mean  values — to 
determine  a  comprehensive  metric,  considering  OPNAV  and  technical  priority  only,  and 
combining  OPNAV  priority  with  DSLOC  estimates.  Clearly,  other  metrics  can  be  easily 
applied  in  the  model  if  required. 
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Common  Size  Factor 

Weighting  Scheme 

1.00  «- 

,  Common  sizing 

-  inputs  and 
using  weights 

-  to  obtain  EMV  — 

*  0.10 

2.00 

2.00 

uujetuves 

OPNAV  Weight 

50% 

35% 

35% 

15% 

15% 

SMEVAand 
DSLOC  Only 

All  Weighted 
&  Common 
Sized 

OPNAV & 
Technical 
Priority 

OPNAV 
Priority  & 
DSLOC 

Capability 

High 

Mid 

Low 

SME  Mean 
Value  Added 

Technical 

Priority 

OPNAV 

Priority 

DSLOC 

Complexity 

Cost 

Simulation 

Technical 
Priority  H-L 

OPNAV 
Priority  H-L 

EMV  Score 

EMV  Score 

EMV  Score 

EMV  Score 

Objective 

Used 

CV 

Capability  1 

43.00 

3 

2 

278 

21 

22 

321.00 

37.68 

43.00 

35.90 

37.68 

0.1834 

Capability  2 

28.00 

2 

1 

541 

22 

23 

569.00 

42.24 

45.00 

50.05 

42.24 

0.0903 

Capability  3 

13.00 

1 

3 

58 

23 

21 

71.00 

19.78 

44.00 

23.90 

19.78 

0.0719 

Capability  4 

40.33 

5 

4 

635 

19 

20 

675.33 

48.04 

39.00 

51.75 

48.04 

0.0576 

Capability  5 

17.67 

7 

7 

134 

17 

17 

151.67 

21.07 

34.00 

23.70 

21.07 

0.0407 

Capability  6 

35.67 

4 

6 

392 

20 

18 

427.67 

37.60 

38.00 

37.60 

37.60 

0.1568 

Capability  7 

50.67 

6 

5 

549 

18 

19 

599.67 

48.05 

37.00 

46.45 

48.05 

0.0481 

Capability  8 

47.67 

10 

11 

675 

Data 

14 

13 

722.67 

48.41 

27.00 

46.75 

48.41 

0.0689 

Capability  9 

Data 

19.00 

8 

8 

109 

Redacted 

16 

16 

128.00 

20.07 

32.00 

21.45 

20.07 

0.0382 

Capability  10 

Kedactea 

33.67 

11 

9 

189 

13 

15 

222.67 

26.80 

28.00 

24.45 

26.80 

0.0603 

Capability  1 1 

20.00 

9 

10 

88 

15 

14 

108.00 

18.78 

29.00 

18.40 

18.78 

0.0707 

Capability  12 

27.67 

14 

12 

159 

10 

12 

186.67 

21.85 

22.00 

19.95 

21.85 

0.1137 

Capability  13 

27.67 

12 

13 

159 

12 

11 

186.67 

22.15 

23.00 

18.95 

22.15 

0.0408 

Capability  14 

44.67 

13 

14 

523 

11 

10 

567.67 

40.24 

21.00 

36.15 

40.24 

0.0956 

Capability  15 

17.67 

20 

15 

134 

4 

9 

151.67 

14.77 

13.00 

15.70 

14.77 

0.0603 

Capability  16 

17.67 

21 

16 

134 

3 

8 

151.67 

14.17 

11.00 

14.70 

14.17 

0.0816 

Capability  17 

34.33 

19 

17 

328 

5 

7 

362.33 

27.10 

12.00 

23.40 

27.10 

0.0454 

Capability  18 

17.67 

22 

18 

134 

2 

6 

151.67 

13.27 

8.00 

12.70 

13.27 

0.0372 

Capability  19 

14.00 

23 

19 

81 

1 

5 

95.00 

9.54 

6.00 

9.05 

9.54 

0.0340 

Capability  20 

17.67 

15 

20 

134 

9 

4 

151.67 

14.77 

13.00 

10.70 

14.77 

0.0164 

Capability  21 

22.67 

18 

21 

144 

6 

3 

166.67 

15.67 

9.00 

10.20 

15.67 

0.0454 

Capability  22 

11.67 

16 

22 

180 

8 

2 

191.67 

13.38 

10.00 

11.00 

13.38 

0.0468 

Capability  23 

10.00 

17 

23 

95 

7 

1 

105.00 

9.23 

8.00 

5.75 

9.23 

0.0603 

Starting  with  23 
capabilities,  more  to 
be  added  later 

High,  likely,  low  values 
for  running  risk 
simulations 


Technical  and 
OPNAV  priorities 


Complexity 
modeled  using 
DSLOC 


Intermediate  computations:  risk 
simulation  assumptions,  re-adjusted 
priorities,  expected  military  score,  cost 
-based  risk  coefficients 


Input  Assumptions 


Portfolio  Optimization 

Static  and  dynamic  optimization  runs  were  executed  based  on  four  different  EMV 
measures  for  multi-criteria  optimization  to  determine  the  best  allocation  and  selection  of 
capabilities  given  a  nominal  $150  million  budget  constraint  for  each  ACB.  Figure  2  shows 
the  portfolio  optimization  model,  in  which  we  have  the  23  capabilities  listed  (clearly,  we  can 
add  as  many  additional  capabilities  as  required,  as  long  as  we  have  valid  data  and 
assumptions  for  each  capability).  The  EMV  values  (column  C  in  Figure  2)  show  the  value  of 
the  composite  EMV  metric,  depending  on  the  calculation  method  chosen,  the  EMV  value 
using  risk-simulated  cost  estimates  (column  D),  and  EMV  value  using  an  estimate  of  risk 
(columns  E  and  F)  for  each  component  (for  the  initial  study,  we  impute  the  risk  as  the 
budget  cost  overrun  and  variability,  whereas  we  can  add  additional  variables  and  measures 
of  risk  later,  as  required).  Finally,  there  is  a  column  of  decision  variables,  or  “go”  and  “no-go” 
variables  (column  H),  which  are  the  decisions  that  are  being  optimized,  such  that  the  total 
portfolio  EMV  objective  (cell  C28)  is  maximized.  The  total  cost  of  the  portfolio  is  also 
computed  (cell  D28),  and  the  portfolio  is  run  subject  to  a  cost  constraint  of  less  than  or  equal 
to  $150  million  (cell  D29).  For  future  applications,  we  can  add  to  the  existing  optimization 
model  by  also  considering: 

•  Additional  Capabilities  as  required,  beyond  the  initial  list  of  23, 

•  Optimization  and  selection  of  Components,  instead  of  Capabilities, 

•  New  and  alternate  EMV  metrics  beyond  the  four  EMV  estimates  currently 
used, 

•  Additions  of  cross-constraints  such  as  mutually  exclusive  projects  and 
capabilities,  and  the  dependence  of  one  capability  on  another, 
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•  Inclusion  of  additional  constraints  such  as  full-time  equivalences,  facilities, 
etc.,  and 

•  Estimates  of  technical  or  schedule  risk. 
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1  EMV  Method 

Selection 


3 

Capability 

EMV 

Cost 

Risk$ 

Risk  % 

4 

1  Capability  1 

37.6800 

$6.91 

18  34% 

5 

2  Capability  2 

42.2350 

$3.81 

9.03% 

6 

3  Capability  3 

19.7800 

$1.42 

7.19% 

7 

4  Capability  4 

48  0417 

$2.77 

5  76% 

8 

5  Capability  5 

21.0733 

Data 

$0.86 

4.07% 

9 

6  Capability  6 

37  6033 

Redacted 

$5.90 

15.68% 

10 

7  Capability  7 

48  0483 

$2.31 

4.81% 

11 

8  Capability  8 

48  4083 

$3.34 

6  89% 

12 

9  Capability  9 

20.0650 

$0  77 

3  82% 

13 

10  Capability  10 

26  7983 

$1.62 

6.03% 

14 

11  Capability  11 

18.7800 

$1.33 

7  07% 

15 

12  Capability  12 

21.8483 

$2  48 

11.37% 

16 

13  Capability  13 

22  1483 

$0.90 

4  08% 

17 

14  Capability  14 

40.2383 

$3  85 

9.56% 

18 

15  Capability  15 

14. 7733 

$0.89 

6.03% 

19 

1 6  Capability  1 6 

14.1733 

$1.16 

8.16% 

20 

17  Capability  17 

27.0967 

$1.23 

4.54% 

21 

18  Capability  18 

13.2733 

$0.49 

3  72% 

22 

19  Capability  19 

9  5350 

$0.32 

3.40% 

23 

20  Capability  20 

14. 7733 

$0.24 

1.64% 

24 

21  Capability  21 

15  6733 

$0.71 

4.54% 

25 

22  Capability  22 

13.3833 

$0  63 

4.68% 

26 

23  Capability  23 

9  2250 

$0  56 

6.03% 

28 

Total 

584  66 

$645.00 

$12.49 

29 

Constraints: 

MAX 

$150.00  | 

Go  and  No-Go  decisions  on 
portfolio  optimization 

Benefits,  cost,  risk,  EMV  are 
all  considered...  additional 
constraints  and  variables 
can  be  easily  added 

Changing  Constraints 

(Markowitz  Efficient  Frontier) 
Min 


Max 

Step 


Sharpe  Ratio 


46.8161 


Cost-Value  Correlation 
0.3592 


Constraints  can  be  set  (e.g., 
budget,  capability  count,  FTE, 
priorities,  and  new  constraints 
can  be  quickly  added) 


31  All  Manual  Permutations:  2.585E+22 

32  Risk  Simulator’s  Optimization  Profile: 

33  EMV  ACB14  Cost  Runs  all  23  Capabilities  to  find  the  best  combination  of  capabilities  in  ACB  1 4  subject  to 

34  $1 50M  Budget  Constraint,  Max  EMV  (Choose  EMV  Type) 

35  Instructions:  Reset  all  selections  to  1  (H4.H26),  select  EMV  type,  click  on  Run  Optimization! 

36  1  Corresponding  ROV  Modeler  file:  01  Discrete  Optimization  EMV  (Functional  Complexity  Only)  ACB14.ro 

37  Corresponding  ROV  Modeler  file:  02  Discrete  Optimization  EMV  (Weighted  and  Common  Sized)  ACB14.ro 

38  Corresponding  ROV  Modeler  file:  03  Discrete  Optimization  EMV  (Weighted)  ACB14.ro 

Figure  1.  Portfolio  Optimization  Model 

Figure  3  illustrates  the  portfolio  optimization  and  capability  selection  setup  using  the 
Risk  Simulator  software.  It  shows  Static  and  Dynamic  Optimization  routines  run  on  multiple 
decision  variables  and  constraints.  It  also  shows  the  exact  specifications  of  the  model. 
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Figure  2.  Optimization  Model  Setup  in  Risk  Simulator 
Expected  Military  Value  (EMV) 

The  next  step  in  the  analysis  was  to  apply  the  combination  of  4  EMV  methods  to 
obtain  the  portfolio  of  options  results  and  recommendations  for  sequencing  capability 
insertions  for  ACB  14,  ACB  16,  and  ACB  18.  Figures  4  through  7  show  the  details  of  each 
portfolio  optimization  run  and  their  corresponding  capability  set  selected.  The  specification 
of  each  optimization  depends  on  the  EMV  that  is  selected.  For  instance,  the  first  set  of 
results  below  is  run  based  on  using  the  mean  of  the  subject-matter  experts’  (SME)  value- 
added  estimates,  software  lines  of  code  (DSLOC),  and  OPNAV  and  technical  priorities,  and 
all  these  variables  are  combined  through  weighting  and  common-sizing  the  averages.  The 
portfolio  optimization  is  run  to  determine  the  best  capabilities  to  select  to  maximize  the  total 
EMV  for  the  portfolio,  while  at  the  same  time  maximizing  the  EMV,  subject  to  the  $150 
million  budget  constraint.  These  results  indicate  a  multi-criteria  optimization  routine,  in  which 
various  objectives  or  EMVs  are  used  in  the  portfolio-selection  process. 
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Expected  Military  Value:  SME  Mean  Value  Added,  DSLOC  Complexity,  Common  Sized,  Weighted  OPNAV/Technical  Priorities  I  ▼ 


Capability 

EMV 

Cost 

1  Capability  1 

37.6800 

2  Capability  2 

42.2350 

3  Capability  3 

19.7800 

4 

Capability  4 

48.0417 

5 

Capability  5 

21.0733 

6  Capability  6 

37.6033 

Data 

Redacted 

7 

Capability  7 

48.0483 

8  Capability  8 

48.4083 

9 

Capability  9 

20.0650 

10 

Capability  10 

26  7983 

11 

Capability  1 1 

18  7800 

12  Capability  12 

21.8483 

13 

Capability  13 

22  1483 

14 

Capability  14 

40  2383 

15 

Capability  15 

14.7733 

16 

Capability  16 

14.1733 

17 

Capability  17 

27.0967 

18 

Capability  18 

13.2733 

19 

Capability  19 

9.5350 

20 

Capability  20 

14. 7733 

21 

Capability  21 

15.6733 

22 

Capability  22 

13  3833 

23 

Capability  23 

9.2250 
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Figure  3.  ACB  14  Optimization  Results 
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Sharpe  Ratio  36.2059 


Changing  Constraints 

(Markowitz  Efficient  Frontier) 


Min 
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Step 


Cost-Value  Correlation 
0.3592 


Figure  4.  ACB  14  Optimization  Results  II 
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Sharpe  Ratio  34.9815 


Figure  5.  ACB  14  Optimization  Results  III 


Sharpe  Ratio  40.3186 


Figure  6.  ACB  14  Optimization  Results  IV 

Figure  8  summarizes  the  capabilities  chosen  based  on  each  of  the  four  EMV 
approaches,  and  the  resultant  recommendations  for  implementation.  Specifically,  the  last 
column  shows  the  optimal  decision  based  on  a  portfolio  of  options  of  decisions.  For 
instance,  the  following  capabilities  should  be  considered  as  optimal  for  ACB  14:  Capabilities 
4,7,  10,  11,  13,  14,  15,  16,  17,  and  18.  Using  the  multi-criteria  optimization,  the  analysis 
does  not  simply  rely  on  a  single  estimate  of  EMV,  but  is  able  to  employ  the  data  from 
multiple  facets  and  triangulate  the  best  course  of  action.  Figure  8  shows  the  results.  It 
illustrates  that  certain  components  are  always  selected  regardless  of  the  EMV  metric  used, 
providing  a  higher  level  of  comfort  in  the  analysis,  and  these  are  the  components  we 
recommend  (last  column  in  the  figure).  Further,  there  are  multiple  components  that  3  out  of 
4  of  the  optimization  routines  suggest  executing;  in  most  cases,  these  are  not  considered 
the  top  10  components,  but  nonetheless  important  in  the  current  ACB  14.  These  4  EMV 
choices  provide  a  view  on  the  Analysis  of  Alternatives.  The  results  of  this  analysis 
postulates  which  components  are  considered  the  top  10  and  which  are  not,  while  still  being 
critical  in  the  ACB  14  portfolio.  Using  these  four  EMV  options,  we  have  four  optimal 
portfolios,  and  we  can  quickly  determine  the  best  Course  of  Action  (shown  as  the  last 
column  in  Figure  8). 
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Discrete  Optimization  EMV  (OPNAV  and  Technical  Priorities  Only)  ACB14 


SMEMVA,  DSLOC,  SMEMVA,  DSLOC, 


Technical  &  OPNAV  Priority 


Priorities,  Sized  Priorities,  Sized  SMEMVA,  DSLOC  OPNAV  Priorities  &  DSLOC 
Constraint  $150M  $150M  Top  10  $150M  S150M  S150M 

Capability 


Recommendation 
Option  1 
$134 


Recommendation 
Option  2 
146 


Figure  7.  ACB  14  Portfolio  of  Options  Results 
Sequencing  of  ACB  14,  ACB  16,  ACB  18 

The  next  step  in  the  analysis  was  performing  a  sequential  compound  portfolio  option 
by  examining  ACB  16  and  ACB  18.  In  other  words,  based  on  the  analysis  for  ACB1  4,  the 
budget  of  no  more  than  $150  million  will  be  spent  on  1 1  capabilities,  with  the  remaining  12 
capabilities  still  available  for  future  execution.  So,  with  another  $150  million  budget  in  ACB 
16,  the  portfolio  optimization  was  rerun  with  the  truncated  list  of  available  capabilities,  and 
the  results  shown  in  Figure  9  were  obtained.  The  analysis  considers  capabilities  that  may 
not  be  available  until  later  ACBs  by  simply  including  them  in  the  process  beginning  with  the 
earliest  ACB  for  which  they  are  ready  for  integration.  The  optimization  is  repeated  based  on 
each  of  the  four  multi-criteria  objectives  and  provides  a  list  of  recommended  capabilities  to 
execute  (highlighted  box  in  Figure  10  shows  the  recommended  components  in  this  ACB). 
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Figure  8.  ACB  16  Optimization  Results 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


750 


Discrete  Optimization  EMVACB16 


SMEMVA,  DSLOC, 
Priorities,  Sized 
$150M 


A  CBM 
A  CBM 


GO 


A  CBM 
A  CB14 
A  CB14 
A  GB14 
A  CB14 
A  CB14 


GO 

GO 

GO 


Technical  & 

SMEMVA,  DSLOC  OPNAV  Priorities 
$150M  $150M 


A  CB14 
A  CB "?  4 

A  CB14 
A  CB "?  4 
A  CB "?  4 
A  CB "?  4 
A  CB "?  4 
A  CB "?  4 


GO 

GO 


ACB14 

ACB14 


GO 


ACB14 

ACB14 

ACB14 

AGB14 

ACB14 

ACB14 


GO 

GO 

GO 


OPNAV  Priority 
&  DSLOC 
$1S0M 


GO 

GO 

GO 

GO 

1 

A  CB  ?  4 

A  CB "?  4 

ACB15 

A  CB "?  & 

GO 

GO 

GO 

1 

A  CB  ■?  4 

A  CB  1 4 

ACB14 

A  CB "?  4 

A  CB  ■?  4 

A  CB "?  4 

GO 

ACB14 

A  CB  1 4 

A  CB14 
A  CB "?  4 


GO 


A  CB "?  4 
A  CB  1 4 
A  CB "?  4 
A  CB "?  4 
A  CB "?  4 
A  CB "?  4 


GO 

GO 

GO 


Recommendation 
Option  1 
$131 


Figure  9.  ACB  16  Portfolio  of  Options  Results 

The  analysis  continued  with  the  portfolio  optimization  on  ACB  18.  The  highlighted 
boxes  in  Figures  1 1  and  12  show  the  recommended  component  to  execute  in  ACB  18,  and 
in  this  case  it  is  Capability  2.  Capability  2  holds  the  highest  OPNAV  priority,  but  when  the 
analysis  includes  the  other  inputs  into  the  EMV  metrics  (DSLOC,  Technical  Priority, 

Weights,  and  SME  Estimated  Value-Add),  and  considers  the  high  cost  ($126  million)  with 
respect  to  the  allowed  portfolio  budget  ($150  million),  Capability  2  fails  the  selection  for  ACB 
14  and  ACB  16,  and  is  only  recommended  in  ACB  18.  However,  the  analysis  can  consider 
alternate  scenarios  in  which  OPNAV  Priority  is  taken  as  the  most  important  criteria,  and 
Capability  2  is  specified  as  a  “must-have”  in  ACB  14.  The  analysis  was  then  rerun  to 
determine  the  optimal  portfolio  given  this  new  requirement.  The  results  of  that  run  are 
documented  in  the  next  section.  The  report  next  illustrates  the  effects  on  EMV  of  selecting 
Capability  2  in  ACB  14  through  to  the  year  2025.  The  report  then  continues  with  the  Efficient 
Frontier  analysis  to  show  what  additional  components  should  be  added  in  each  ACB  if 
additional  budget  is  allocated  (e.g.,  what  if  the  budget  were  extended  to  $175  million  or  $200 
million,  and  so  forth,  to  determine  at  what  point  perhaps  more  critical  components  would 
have  been  selected).  The  optimization  analysis  is  highly  flexible  to  accommodate  such 
alternate  scenarios  and  requirements. 
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Figure  10.  ACB  18  Optimization  Results 
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Figure  1 1 .  ACB  1 8  Portfolio  of  Options 


Alternate  Scenario:  Capability  2  “Must-Have” 

The  optimization  model  and  approach  used  is  highly  adaptive  and  flexible.  The  next 
analytic  step  conducted  was  the  specification  of  one  or  more  components  as  a  "must  have," 
specifically,  Capability  2  was  set  as  a  mandatory  capability  for  inclusion  in  ACB  14.  Figure 
13,  illustrates  the  generation  of  an  alternate  scenario  by  applying  OPNAV’s  Priority  1  on 
Capability  2  as  a  “must-have”  in  the  portfolio  selection  and  by  identifying  what  other 
capabilities  should  be  inserted  in  each  ACB  examined  in  such  a  scenario.  When  the 
integration  budget  is  constrained  to  $150  million,  if  a  significant  portion  of  it  is  allocated  to 
Capability  2,  then  only  a  little  is  left  over  for  other  components.  Figure  13  shows  what  these 
components  are  (i.e.,  on  the  last  column  of  Figure  13,  the  decision  variable  set  as  1 
indicates  a  “go”  decision,  whereas  0  indicates  a  “no-go”). 
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|  Expected  Milrtar/ Value:  SHE  Mean  Value  Added,  DSLGC  Complexity,  Common  Sized,  Weighted  OPNAV/ Technical  Prrorrties 


13 


CapaM/ly  _ EM\/ _  Cosf  Rlsfc  St 


f 

Capability  1 

37.6300 

|  S6.91 

Capability  2 

42  2350 

S3.  31 

3 

Capability  3 

19.  7300 

S1.42 

4 

Capability  4 

43  0417 

S2.  77 

5 

Capability  5 

21.0733 

SO.  36 

6 

Capability  6 

37.6033 

S5.  90 

7 

Capability  7 

43.  0433 

S2.31 

3 

Capability  8 

48.  4033 

Data 

CZ>  A  f-J  JJ  A  4-  A  Jfj 

S3.  34 

9 

Capability  9 

20.  0650 

rxC  U  cdlCJ  Lc  U 

SO.  77 

io 

Capability  IO 

26  7983 

SI. 62 

11 

Capability  1 1 

13.  7300 

SI. 33 

12 

Capability  1 2 

21.3433 

S2.  48 

13 

Capability  13 

22  1483 

SO  90 

14 

Capability  1 4 

40.  2333 

S3.  35 

15 

Capability  15 

14.  7733 

SO.  39 

16 

Capability  1 6 

14.  1733 

SI.  16 

17 

Capability  1 7 

27  0967 

SI.  23 

13 

Capability  13 

13.2733 

SO.  49 

19 

Capability  19 

9.  5350 

SO.  32 

20 

Capability  20 

14.  7733 

SO.  24 

21 

Capability  21 

15.6733 

SO.  71 

22 

Capability  22 

13.  3333 

SO  63 

23 

Capability  23 

9.  2250 

I  SO.  56 

Total 

115.  56 

SI 49.  OO 

S4.  40 

Constraints : 

MAX  | 

SI  50.00  \ 

Sharpe  Ratio 

26  2374 

FlisH  % 
13.  34% 
9.  03 % 
7  19% 

5.  76% 
4.07% 
15.  63% 
4.31% 

6.  39% 
3  82% 
6.  03% 
7.07% 

11.37% 
4.  08% 
9.  56% 
6.  03% 

3.  16% 

4.  54% 
3  72% 
3  40% 
1 .  64% 
4.  54% 
4.  68% 
6  03% 


Selection 


Capability  2 
"Must  Have" 


Figure  12.  Capability  2  “Must-Have”  in  ACB  14 

Aggregate  EMV 

By  computing  EMV  accrued  for  each  ACB,  the  analysis  can  then  track  the 
aggregation  of  value  in  terms  of  EMV  available  by  ship-year  based  on  the  year  of  installation 
of  each  ACB  in  an  individual  ship.  This  approach  permits  representing  the  total  capability 
available  to  the  fleet  as  a  single  number.  Figure  14  shows  the  ship  availabilities  for  ACB 
insertions.  Using  this  ship  schedule  and  availability  for  ACB  insertion  and  applying  the 
optimal  EMV  values,  the  aggregate  EMV  values  through  the  year  2025  become  known 
(Figure  15).  The  second  curve  in  Figure  15  demonstrates  that  EMV  over  time  is  marginally 
reduced  by  requiring  Capability  2  to  be  included  in  ACB  14.  Figure  15  also  shows  the  “catch 
up”  effect  of  the  ACB  process.  Even  though  the  introduction  of  a  capability  might  be  delayed 
from  one  ACB  to  the  next,  the  total  number  of  ships  possessing  the  capability  will  become 
the  same  after  the  fourth  ACB  period  if  the  delayed  capability  is  included  in  the  next  update. 
An  alternative  to  consider  is  to  maintain  the  $150  million  budget  across  all  ACBs,  but  at  the 
same  time  increase  the  ACB  14  budget  to  include  a  “special  insertion  budget”  to  cover 
Capability  2  and  maintain  the  portfolio  as  suggested  previously. 
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Figure  13.  ACB  14  Ship  Availabilities 


Aggregation  of  Relative  EMV 


Capability  Realization  by  ACB  Introduced 


Figure  14.  ACB  14  Aggregate  EMV 


Monte  Carlo  Risk  Simulation 


Monte  Carlo  Risk  Simulation  of  cost  estimates  is  used  to  determine  the  risk  of  budget 
and  cost  overruns  (project  timing  overruns  can  also  be  determined  if  required).  A  sample  of 
the  simulated  risk  analysis  results  is  shown  in  Figure  16.  While  capability  selection  is  the  key 
question  addressed  in  this  study,  the  risk  analysis  results  are  necessary  and  support  some 
of  the  optimization  analysis.  For  future  modeling  and  decision-analysis  work,  the  proper 
determination  of  appropriate  risk  measures,  potentially  including  cost,  schedule,  technology, 
and  other  risks,  would  be  appropriate  and  beneficial.  The  analyst  can  model  all  these 
uncertainties  using  the  Risk  Simulator  software  tools.  Instead  of  relying  on  single-point 
estimates  for  cost  and  scheduled  completion  times,  distributions  of  cost  and  time  through 
expert  estimates,  comparable  historical  data,  and  expectations  of  high,  most-likely,  and  low 
estimates  for  each  input  should  be  employed.  The  analytic  method  then  specifies  that  these 
values  be  simulated  thousands  of  times  with  the  software  to  generate  all  possible  outcomes 
and  scenarios,  and  the  results  are  then  interpreted  to  examine  the  risks  inherent  in  each 
ACB  insertion.  Applying  this  method  to  the  first  analysis,  the  results  indicate  that  although 
the  expected  total  cost  is  $150  million,  there  is  83.30%  chance  that  the  budget  will  be 
exceeded.  In  fact,  to  be  99%  sure  that  there  is  sufficient  money  to  cover  the  potential  cost- 
creep,  the  budget  would  have  to  be  increased  to  $171  million,  indicating  the  need  of  a  $21 
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million  cushion.  Similarly,  the  analyst  can  apply  the  methodology  to  determine  the  probability 
of  the  occurrence  of  schedule  overruns.  Other  risks,  such  as  technology  risk,  may  be 
expressed  in  other  ways  to  provide  inputs  to  the  simulator  software. 


Risk  Simulation  of  Cost 


'Screen  shots  from  Risk  Simulator  software 
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Figure  15.  Risk  Analysis  on  Cost  for  ACB  14 


Portfolio  Optimization’s  Efficient  Frontier 

The  portfolio  Efficient  Frontier  analysis  determines  multiple  scenarios  of  increasing 
budget  (i.e.,  if  the  $150  million  budget  were  increased  to  $200  million,  or  $250  million,  or 
$300  million,  and  so  forth,  in  various  increments,  what  will  the  optimal  portfolio  look  like; 
what  capabilities  should  be  added  or  replaced;  and  what  are  the  impacts  on  EMV?).  This 
analysis  provides  useful  input  for  deliberations  with  the  sponsor  early  in  the  budget- 
development  process  and  yields  data-driven  sets  of  alternatives  for  various  levels  of 
funding. 

Running  the  optimization  procedure  yields  an  optimal  portfolio  of  projects  in  which 
the  constraints  are  satisfied.  This  represents  a  single  optimal  portfolio  point  on  the  Efficient 
Frontier — for  example,  Portfolio  B  on  the  Efficient  Frontier  chart  in  Figure  17.  Then,  by 
subsequently  changing  some  of  the  constraints — for  instance,  by  increasing  the  budget — the 
analyst  can  rerun  the  optimization  to  produce  another  optimal  portfolio  given  these  new 
constraints.  Therefore,  a  series  of  optimal  portfolio  allocations  can  be  determined  and 
graphed.  This  graphical  representation  of  all  optimal  portfolios  is  called  the  portfolio’s 
Efficient  Frontier.  At  this  juncture,  each  point  represents  a  portfolio  allocation.  For  instance, 
Portfolio  B  might  represent  capabilities  1, 2,  5,  6,  7,  8,  10,  15,  and  so  forth,  while  Portfolio  C 
might  represent  capabilities  2,  6,  7,  9,  12,  15,  and  so  forth — each  resulting  in  different  EMV, 
tactical,  military,  or  comprehensive  scores,  and  portfolio  returns.  It  is  up  to  the  decision¬ 
maker  to  decide  which  portfolio  represents  the  best  decision  and  if  sufficient  resources  exist 
to  execute  these  projects.  Typically,  in  an  Efficient  Frontier  analysis,  a  decision-maker  would 
select  projects  for  which  the  marginal  increase  in  benefits  is  positive,  and  the  slope  is  steep. 
In  the  next  example,  that  decision-maker  would  rather  select  Portfolio  D  rather  than  Portfolio 
E,  as  the  marginal  increase  is  negative  on  the  y-axis  (e.g.,  EMV).  That  is,  spending  too 
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much  money  may  actually  reduce  the  overall  EMV;  hence,  this  portfolio  should  not  be 
selected.  Also,  in  comparing  Portfolios  A  and  B,  a  decision-maker  would  be  more  inclined  to 
choose  B,  as  the  slope  is  steep  and  the  same  increase  in  budget  requirements  (x-axis) 
would  return  a  much  higher  percentage  EMV  (y-axis).  The  decision  to  choose  between 
Portfolios  C  and  D  would  depend  on  available  resources,  and  the  decision-maker  must 
decide  if  the  added  benefits  warrant  and  justify  the  added  budget  and  costs.  Figures  18 
through  22  illustrate  the  results  from  the  Efficient  Frontier  analysis  by  changing  the  budget 
constraint  from  $150  million  to  $300  million  by  incrementing  it  $25  million  in  each  step. 


Figure  16.  The  Theory  of  Portfolio  Efficient  Frontier 
Portfolio  Optimization:  Efficient  Frontier  Analysis 


ACB  14  Markowitz  efficient  frontier  (x-axis  is  budget  amount, 


and  y-axis  is  total  expected  military  value) 

“Screen  shots  from  ROV  Optimizer  software 

Figure  17.  Portfolio  Efficient  Frontier 
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Discrete  Optimization  EMV  (OPNAV  and  Technical  Priorities  Only)  ACB14 
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Figure  18.  Efficient  Frontier  with  Additional  Budget  Allocation  I 


Optimization  EMV  (SMEMVA  DSLOC  Weighted,  Common  Sized,  Priorities)  ACB14  Efficient 
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Figure  19.  Efficient  Frontier  with  Additional  Budget  Allocation 


Discrete  Optimi; 


Figure  20.  Efficient  Frontier  with  Additional  Budget  Allocation  III 


Discrete  Optimization  EMV  (OPNAV  DSLOC  Only)  ACB14  Efficient  Frontier 

Constraint  $150M  $175M  $200M  $225M 

Capability  $146M  $175M  $200M  $2<X>M 
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Figure  21 .  Efficient  Frontier  with  Additional  Budget  Allocation  IV 


Optimization  with  a  Risk  Constraint 

Figure  23  shows  an  example  optimization  run  in  which  we  can  set  cost  as  the 
stochastic  constraint.  That  is,  seeing  that  cost  overruns  typically  occur  in  development,  we 
can  set  the  risk  simulation  optimization  combination  model  such  that  we  want  a  portfolio 
where  there  is  a  90%  probability  that  the  $150  million  budget  is  not  exceeded.  In  this  sample 
run,  we  see  that  this  can  be  accomplished  by  replacing  Capability  9  with  Capability  23,  at  a 
lower  cost,  thereby  still  creating  the  maximum  EMV  possible  while  maintaining  a  90% 
probability  that  total  portfolio  cost  will  be  under  the  required  $150  million  budget  constraint. 
Alternatively,  as  shown  previously  in  Figure  16,  if  the  optimal  portfolio  is  still  desirable,  then 
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the  $150  million  budget  needs  to  account  for  a  potential  overrun  of  $22  million.  That  is, 
there  is  a  99%  probability  that  the  total  portfolio  budget  will  be  under  $172  million.  There  is 
clearly  a  risk-value  tradeoff  occurring  in  this  situation;  the  higher  the  probability  of  lower 
budget  overruns,  the  lower  the  anticipated  EMV.  This  tradeoff  is  also  seen  very  clearly  in  the 
Efficient  Frontier  analysis,  in  which  the  results  demonstrate  that  the  higher  the  budget 
allocation,  the  greater  the  EMV.  The  final  step  in  the  analysis  was  a  sample  run  with  ACB 
14,  16,  and  18  based  on  the  EMV  applying  all  input  assumptions,  and  with  the  additional 
contingent  constraint  the  total  budget  used  will  not  exceed  $150  million  for  each  ACB  for  at 
least  85%  of  the  time.  The  results  are  shown  in  Figures  24,  25,  and  26  for  each  of  the  ACBs 
sequenced.  The  resulting  budget  overrun  risks  are  depicted  in  Figure  27.  When  this 
constraint  is  applied,  the  analysis  yields  a  different  portfolio  selection  for  each  ACB,  and  the 
probability  of  not  exceeding  the  requisite  $150  million  budget  becomes  97.99%,  90.90% 
and  99.99%  for  the  three  ACB  years.  Running  the  model  with  Capability  2  as  mandatory  in 
ACB  14  and  applying  the  risk  constraint,  generates  yet  another  set  of  selections,  yielding 
confidence  levels  of  not  exceeding  the  budget  of  $150  million.  Figure  28  shows  the  resulting 
aggregate  EMV  across  all  ships,  revised  to  include  the  risk-cost  portfolio.  It  is  clear  that  the 
opportunity  cost  of  applying  the  risk  constraint,  while  measurable,  is  minimal  for  this  case 
and  actually  represents  less  reduction  in  EMV  overall  than  does  the  mandatory  selection  of 
a  capability. 


|  Expected  Military  Value:  SME  Mean  Value  Added,  DSLOC  Complexity,  Common  Sized,  Weighted  OPNAV/Technical  Priorities  (  \ 
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Figure  22.  Cost-based  Risk  Optimization  Alternative 


Figure  23.  Cost-based  Risk  Optimization  for  ACB  14 
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Figure  24.  Cost-based  Risk  Optimization  for  ACB  16 
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Figure  25.  Cost-based  Risk  Optimization  for  ACB  18 
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Figure  26.  Cost-Risk  Probabilities  for  ACB  14, 16,  and  18 


Figure  27.  Aggregate  EMV  for  2014-2025 
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Figures  29  and  30  summarize  the  results  of  all  portfolio  runs  as  a  simple  visual 
matrix  and  list  of  summary  statistics.  The  portfolio  runs’  visual  matrix  of  results  consists  of 
four  columns  of  portfolios:  (1 )  optimization  based  on  the  required  total  budget  of  $150 
million,  assuming  all  costs  are  exact  and  have  no  risk  (Optimal  on  Budget);  (2)  cost  with 
uncertainty  and  risk  that  there  will  be  budget  overruns — with  an  added  constraint  of  a 
portfolio  with  no  more  than  15%  probability  of  a  budget  overrun,  or  85%  probability  or 
higher,  that  the  total  budget  of  $150  million  will  not  be  exceeded  (Optimal  Cost-Risk);  (3) 
Capability  2  is  a  required  component  in  ACB  14  (Capability  2  Must-Have);  and  (4)  Capability 
2  as  a  Must-Have  in  ACB  14,  with  the  added  assumption  of  cost-risk  as  defined  above 
(Capability  2  Cost-Risk).  Similar  ACB  years  are  color-coded  (e.g.,  green  for  ACB  14,  blue  for 
ACB  16,  orange  for  ACB  18,  and  red  for  components  that  are  not  selected  in  these  three 
ACBs,  allowing  for  potential  implementation  later). 

Figure  30  shows  the  summary  key  statistics  of  these  portfolios,  listing  the  number  of 
capabilities  implemented  in  each  ACB  cycle,  the  total  expected  budget  used,  the  total  EMV, 
and  the  probability  that  the  ACB  will  be  under  budget.  Clearly,  the  non-Cost-Risk  portfolios 
bear  higher  total  implementation  costs  with  higher  EMVs  (high  risk  means  high  returns); 
however,  the  probability  of  being  under  budget  is  low,  and  the  probability  the  budget  will  be 
exceeded  is  high.  In  this  First  Phase  analysis,  due  to  the  analysis  being  run  on  only  23 
capabilities,  as  expected,  the  EMVs  are  reallocated  over  time  in  various  amounts  (e.g., 
Capability  2  must-have  will  yield  a  smaller  initial  EMV  due  to  the  higher  cost  and  moderate 
EMV  value  of  executing  Capability  2,  but  the  catch-up  happens  in  the  subsequent  ACBs). 
Figure  30  also  examines  the  risk  distribution  of  the  budget  based  on  the  different  portfolio 
criteria.  For  instance,  we  see  that  if  we  apply  the  value  optimization  without  regard  to  the  risk 
of  cost  overruns  the  median  (or  50th  percentile)  budget  is  $153.2  million,  above  the  budget 
constraint  of  $150  million.  Alternatively,  if  we  consider  the  risk  of  cost  overruns,  the  median 
is  only  $142.9  million,  providing  a  buffer  for  any  overruns.  In  fact,  we  see  that  the  85th 
percentile  is  $146.6  million,  and  the  95th  percentile  is  148.7  million — both  under  the  required 
$150  million.  Further,  there  is  a  97.90%  probability  that  this  portfolio  will  come  in  under  the 
$150  million  budget  (for  the  sake  of  clarity,  these  values  are  highlighted  in  yellow  in  Figure 
31).  To  reduce  and  hedge  the  risk  of  cost  overruns — the  expected  budget  used  is  less  ($139 
million  as  opposed  to  $146  million),  with  a  return  on  EMV  that  is  also  less  (310.98  as 
compared  to  299.74).  Therefore,  to  hedge  and  reduce  the  risk  of  cost  overrun,  the  Navy 
spends  less  and  gets  less.  This  can  be  viewed  as  keeping  some  of  the  budget  aside  for  the 
worst  case  scenario — therefore  leaving  less  money  available  to  invest  in  additional 
capabilities  (the  remaining  statistics  are  fairly  self-explanatory).  One  alternative  to  utilizing 
the  highest  number  of  capabilities,  maximizing  the  EMV,  and  yet  coming  under  budget,  is  to 
consider  strategic  real  options  in  contract  negotiations. 

Decision-makers  should  exercise  caution  in  the  use  of  risk  constraints  to  restrict 
consideration  of  portfolios.  Blind  selection  of  a  risk-limited  portfolio  may  result  in  excessive 
opportunity  cost  if  other  means  exist  to  reduce  risk  in  the  input  data.  For  example,  better 
cost  estimates  in  one  or  more  of  the  components  would  reduce  volatility  in  that  component 
and,  thereby,  make  it  less  likely  to  be  excluded  due  to  a  high  contribution  to  aggregate  risk. 
Similarly,  altering  the  cost  profile  through  risk  mitigation  efforts  in  the  contract  structure 
(caps,  fixed-price  provisions,  etc.)  changes  the  input  and  will  change  the  output  from  the 
model.  By  applying  the  appropriate  risk-mitigation  measures  and  by  rerunning  the  analysis, 
decision-makers  may  provide  a  better  portfolio  selection  than  simply  constraining  the 
analysis  through  applying  a  risk  cap.  Intelligent  use  of  the  toolset  as  a  decision  aid 
maximizes  its  value  to  the  manager. 
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Figure  28.  Summary  of  All  Sequenced  and  Optimized  Portfolios 


ACB  14  +  ACB  16  -#-  ACB  IS 

•rounded  to  the  nearest  0.1 

Total  Capabilities  ACB  1 4 

Total  Capabilities  ACB16 

Total  Capabilities  ACB  18 

EMV  ACB  14 

EMV  ACB1 6 

EMV  ACB18 

Total  Cost  ACB1 4 

Total  Cost  ACB  1 6 

Total  Cost  ACB  1 8 

Total  Spent  on  ACB  1 4-1 8 

Probability  of  Under  Budget  ACB  14 

Probability  of  Under  Budget  ACB  16 

Probability  of  Under  Budget  ACB  18 


Optimal  Optimal  Capability  2  Capability  2 
on  Budget  Cost-Risk  Must-Have  Cost-Risk 


11 

11 

5 

2 

8 

7 

9 

lO 

1 

2 

4 

7 

310.98 

299.74 

115.56 

61.02 

149.87 

151.58 

268.03 

280.96 

42.24 

57.94 

127.93 

151.58 

$146.00 

$139.00 

$149.00 

$129.00 

$141.00 

$129.00 

$150.00 

$137.00 

$126.00 

$95.00 

$141.00 

$129.00 

$413.00 

$363.00 

$440.00 

$395.00 

29.70% 

97.90% 

41.50% 

90.80% 

72.23% 

90.90% 

16.25% 

99.90% 

94.80% 

99.90% 

72.90% 

90.90% 

ACB  14  Median  50tb  Percentile  on  Budget 
ACB  14  Median  85tn  Percentile  on  Budget 
ACB  14  Median  95tn  Percentile  on  Budget 
ACB  16  Median  50th  Percentile  on  Budget 
ACB  16  Median  85tb  Percentile  on  Budget 
ACB  16  Median  95th  Percentile  on  Budget 
ACB  18  Median  50th  Percentile  on  Budget 
ACB  18  Median  85th  Percentile  on  Budget 
ACB  18  Median  95th  Percentile  on  Budget 


$153.20 

$142.90 

$152.60 

$132.30 

^160.22 

$146.60 

$166.60 

$146.30 

$164.30 

$148.70 

$173.90 

$153.50 

$145.40 

$137.90 

$156.50 

$139.50 

$153.50 

$147.30 

$164.80 

$143.30 

$157.80 

$152.70 

$169.20 

$145.30 

$128.90 

$95.30 

$145.10 

$137.90 

$142.90 

$101.30 

$153.50 

$147.30 

$150.20 

$104.30 

$158.40 

$152.70 

Figure  29. 


Summary  of  All  Sequenced  and  Optimized  Portfolios’  Summary  Statistics 
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Introduction  and  Context 


•  Introduction  of  Open  Architecture  (OA)  business  and  technical 
processes  provides  opportunity  to  improve  acquisition 

-  Increased  competition 

-  Shorter  cycle  time 

-  Reduced  total  ownership  and  acquisition  cost 

•  The  AEGIS  Advanced  Capability  Build  (ACB)  process  is  one 
implementation  of  the  OA  approach 


The  ACB  Process 


•  The  ACB  process  provides  for  software  updates  to  ships  within 
the  program  on  a  two-year  cycle 

•  ACBs  are  identified  by  the  first  year  in  which  they  will  be  fielded, 
e.g.,  ACB-14 


•  US  Navy  CGs  and  DDGs  will  be  inducted  into  the  process 
as  they  receive  computing  plant  updates  during  major 
availabilities  that  convert  the  processors  and  networks  to  a 
COTS-based  configuration 


The  hardware  baseline  that  supports  OA  must  be  in  place 
to  begin  execution  of  the  ACB  process 

Once  a  ship  is  inducted,  it  will  receive  the  scheduled 
update  plus  any  previous  updates  (e.g.,  ACB  16  ships 
entering  the  program  will  receive  ACB  14  capabilities  as 
well) 
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The  problem  addressed  in  this  study  deals 
with  risk  and  value 


•  Value  is  realized  through  fielding  of  military  capability  for  the  war-fighter 

•  Risk  is  found  in  uncertainty 

-  Cost  uncertainty  creates  budget  risk 

-  Technology  risk  can  lead  to  schedule  and  budget  risk 

•  This  study  provided  a  pilot  implementation  of  the  Knowledge  Value 
Added  +  Integrated  Risk  Management  method  to  represent  value  and 
risk  to  assist  the  PM  and  sponsor  in  selecting  the  proper  capability  mix 
to  field  in  a  given  ACB 


•  The  problem  space  considered  23  capabilities  to  be  implemented 
through  changes  to  32  software  components  across  three  scheduled 
ACBs  (ACB  14,  16,  and  18) 


•  Given  the  universe  of  desired  capabilities,  the  problem  is  to  select 
those  providing  the  best  value  to  the  war-fighter  for  inclusion  in  a  given 
ACB  subject  to  budget  constraints,  risk  and  uncertainty  of  cost  and 
timing 
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The  study  articulated  a  notional  value  of  military  value 

and  used  powerful  financial  and  analytical  tools 

•  Knowledge  Value  Added  (KVA)  provides  ways  of  representing 
outputs  (value)  in  common  units 

•  Real  Options  provides  tools  to  compare  the  value  stream  of 
various  options  in  rigorous  terms 

•  Integrated  Risk  Management  considers  uncertainties  and 
represents  risk  in  quantitative,  clear  and  defensible  terms 
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As  ships  enter  the  program  and  ACBs  are 
executed,  military  value  is  additive 


Aggregation  of  Relative  EMV 


One  advantage  of  the  ACB  process  is  the 
“catch-up”  effect 


•  The  ACB  process  helps  manage 
risk  by  allowing  the  PM  and 
sponsor  to  delay  introduction  of 
a  particular  capability  until  it  is 
ready  without  waiting  many 
years  for  the  next  cycle 

•  A  delay  until  the  next  ACB  is 
more  acceptable  because  within 
five  years  the  number  of  ships 
with  the  capability  will  be  the 
same 


Capability  Realization  by  ACB  Introduced 
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The  remainder  of  this  presentation  discusses  the 
analysis  and  results 

•  Assumptions  and  constraints 

•  Measurement  of  military  value  (KVA) 

•  Application  of  Real  Options  and  Integrated  Risk  Management 
(IRM)  to  the  selection  of  capabilities 
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Assumptions  and  Constraints 


•  The  study  applied  the  following  initial  assumptions 

-  The  capabilities  were  independent  of  each  other 

-  Capabilities  were  implemented  through  changes  to  modules  within 
the  objective  architecture 

•  Initial  constraints  assumed 

-  A  notional  integration  budget  of  $150  million  per  ACB 

-  All  uncertainty  estimates  for  the  initial  model  were  based  on  cost 


volatility 


•  Future  study  will  include 

-  Effects  of  interdependencies  (nested  options),  correlations, 
diversification 

-  Schedule  risk 

-  Opportunity  cost  and  penalty  costs  of  abandonment 


Measuring  Value 

•  KVA  quantifies  the  value  of  the  knowledge  used  to  produce 
common  units  of  output 

•  Shows  decision  makers  the  benefit  and  cost  of  each  program 
or  project 

•  Measures  how  resources  are  allocated  on  the  volatility  of 
productivity  (e.g.,  ROI) 

•  Providing  ROI  volatility  inputs  to  IRM 

•  Military  value  in  this  study  was  postulated  to  be  represented  by 
capability  provided  to  the  warfighter  measured  in  a  variety  of 
ways 

-  Strategic  importance  as  represented  by  OPNAV  sponsor  priorities 

-  Technical  value  as  represented  by  acquisition  community  priority 

-  Functional  complexity  represented  by  Delivered  Source  Lines  of  Code 
(DSLOC) 

-  Subject  Matter  Expert  evaluation  of  complexity  and  mission  criticality, 
aggregated  from  the  component  level 
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Integrated  Risk  Management  Process 
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List  of  projects 
and  strategies  to 
evaluate 


Start  with  a  list  of  projects 
or  strategies  to  be 
evaluated...  these  projects 
have  already  been  through 
qualitative  screening 
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Framing 
Real  Options 
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...the  relevant  projects 
are  chosen  for  real 
options  analysis  and  the 
project  or  portfolio  real 
options  are  framed... 
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Base  case  projections 
for  each  project 


Time  Series  Forecasting 


. .  .with  the  assistance  of 
time-series  forecasting, 
future  outcomes  can  be 
predicted... 
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Options  analytics, 
simulation,  optimization 


Simulation 


Lattice 


Real  Options  Super 
Lattice  Solver 


. .  .real  options  analytics  are 
calculated  through  binomial  lattices 
and  closed-form  partial-differential 
models  with  simulation... 


O  Develop  static 
financial  models 
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. .  .the  user  generates  a 
traditional  series  of  static  base 
case  financial  (discounted  cash 
flow)  models  for  each  project... 


o 


Portfolio  optimization 
and  asset  allocation 


...stochastic  optimization  is  the 
^  next  optional  step  if  multiple 
m  projects  exist  that  require  efficient 

2  asset  allocation  given  some 
budgetary  constraints...  useful  for 
strategic  portfolio  management. . . 
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Dynamic  Monte 
Carlo  simulation 
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Simulation 
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...Monte  Carlo  simulation  is  added 
to  the  analysis  and  the  financial 
model  outputs  become  inputs  into 
the  real  options  analysis... 


o 


Reports  presentation 
and  update  analysis 


...create  reports,  make 
decisions,  and  do  it  all 
again  iteratively  over  time. . . 
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KVA+RO+IRM  are  a  combination  of  method  and 
toolset  to  assist  the  PM  in  decision  making 


•  Knowledge  Value  Added  (KVA)  is  a  method  that  systematically  expresses  non¬ 
revenue  activities  in  common  units  of  output  to  quantify  value 

•  Real  Options  (RO)  provides  a  way  to  qualitatively  and  quantitatively  evaluate  the 
relative  value  of  various  courses  of  action  under  consideration 

•  Integrated  Risk  Management  combines  KVA  and  RO  with  a  powerful  toolset  to  assist 
the  program  manager  in  the  decision  process 

•  Treatment  of  parameters  as  distributions  permits  rigorous  analysis  in  an  uncertain 
world,  where  instead  of  single  point  estimates,  we  use  ranges  as  inputs 

•  Monte  Carlo  risk  simulation  and  process  models  permit  consideration  of  all  possible 
outcomes  within  a  reasonable  time  period 

•  Disciplined  processes  yield  defensible  results  that  can  be  updated  as  more 
knowledge/  information  is  realized  by  the  program 

•  Risk  simulation,  sensitivity  analysis,  and  forecasting  are  automated  (analyses  are 
efficient,  quick,  consistent,  replicable,  defensible,  and  scalable) 

The  toolset  and  method  provide  a  way  for  the  PM  to  determine  the  relative  merits  of  the 
various  options  available ,  to  make  informed  choices  based  on  value  streams  and  risk ,  and 
then  to  articulate  those  choices  to  the  sponsor  and  the  acquisition  chain  of  command. 


Model  input  assumptions  are 
entered  on  a  data  sheet 


Common  sizing  inputs  and  using  weights 
to  obtain  the  expected  military  value 


I 


Conifftm  Factor 


Capability 

Capabilil- 1 

Capability 

Capability  3 

CacabHit.  4 

CapadiliV 5 

CapabilFT?  6 

capability  7 

Capabii^-3 

car-abntt.  9 

Cwzbm  10 

CapabiliT*  11 

Capability  12 

CapabiM,  13 

CopabiM*  14 

Capabiliti  15 

Capability  1 6 

Capabilrly  17 

Capability  IB 

Capability  is 

Capabiitt,  2D 

CapabiliT-  21 

Capability  22 

capability  £3 

A 


Starting  with  23  capabilities 
(more  to  be  added  later 
when  there  is  sufficient  data) 


A 


Technical  and 
OPNAV  priorities 


High,  most  likely,  low  cost 
estimates  for  running 
thousands  of  simulation  trials 


DSLOC  provides  a  measure 
of  complexity 
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Intermediate  computations:  risk- 
simulation  assumption,  readjusted 
priorities,  expected  military  score 
and  cost-based  risk  coefficients 


Running  the  model  provides  recommended  selections 

ACB  14  sample  results  with  $150M  budget  constraint 


Mrty 


Expected  Value!  SHE  Mfean  Value  Added,  OSLOC  Complexity,  Common  Si£OdP  Weight* d  OPHAV/Tedmkal  Ptjoritieii 

to*  % 


Cap  at  lilt-  1 

Capability  2 

Capability  3 

Capability  4 

Cap-ability  5 

Capability  6 

Capability  7 

Capability  8 

Capability  9 

Capability  ID 

Capability  11 

Capability  12 

Capability  13 

Capability  14 

Capability  15 

Capability  16 

Capability  17 

Capability  13 

Capability  19 

Capability  20 

Capability  21 

Capability  22 

Capability  23 

Max  EMV 


TofaJ 

Constraints. 


Re&et 


BMV 

COJf 

37.88 

42.24 

19.7$ 

48.04 

21.07 

37.60 

o 

o 

on 

48.05 

48.41 

i— 1- 

2  0.07 

O 

2680 

QJ 

18.78 

r+ 

Ckl 

2185 

LLP 

22.15 

ZE 

0) 

O  ! 

40  24 

i  j  77  X 

1  1 

m.  s  t 

14.17 

ui . 

n 

27.10 

r+ 

fD 

13.27 

Q_ 

9.54 

14.77 

15.67 

13.38 

9.23 

|  3)4.5?  | 

£ 150.00 

MAX 

S  150.00 

Starting  with  23  capabilities  (more  to  be 
added  later  when  there  is  sufficient  data) 


ms*  5 
$6.91 
$3.81 
$1.42 
$ 2.77 
$08$ 
$5.90 
$23  1 
$3.34 
$0.7? 
$382 
$333 
$2.48 
$ 0.90 
S3. 85 
$0.89- 

$1J$ 
$1  23 
$0.49 
$032 
$0  24 
$0.71 
$0  83 
$0.5$ 

$8. 18 
< - 


_  Selection  of  EMV 
calculation  method 


£mc»m 


18.34% 

9.03% 

7.19% 

5  76% 

4.07% 

15.68% 

4.81% 

6.89% 

3.82% 

6.03% 

7.07% 

1137% 

4.08% 

9.5$% 

8.1$% 

4.54 % 

3.72% 

540% 

164% 

4.54% 

4  68% 

$.03% 

0.0000 

0.0000 

doooo 

t.0000 

toooo 

0.0000 

1.0000 

0.0000 

1.0000 

1.0000 

t.0000 

ooooo 

1.0000 

1.0000 

t.0000 

t.oooo 

1.0000 

f  .0000 

o.oooo 

0.0000 

0.0000 

0.0000 

0.0000 

_  Go  or  No-Go 
decisions  in  the 
portfolio  selection 


Benefits  (EMV), 
Cost,  Risk  are 
considered 


Constraints  can  be  set  (budget, 
capability  count,  FTE,  priorities,  etc) 
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Portfolio  optimization  analysis  gives  a  set  of  solutions 

- B 


R]  Optimization  Summary 


Optimization  is  used  to  allocate  resources  where  the 
results  provide  the  max  returns  or  the  min  cost/risks. 
Uses  include  managing  inventories,  financial 
portfolio  allocation,  product  mix,  project  selection, 
etc. 


OptLmlzadan 


Objective  Method  !  Constraints  j  Statistics  Decision  Variables 


05D52E<=  150(50529) 
□  5H52B  *;=  10  (5H529) 


Add 


Chance 


KJ  OptimijfllaOn  Symrtifljy 


Opfen-rzstson  is  used  ta  allocate  rf5f^rc«  where  the 
results  peovadts  the  n-.iv  returns  -or  the  mm  i-cal'FiFks 
Uses  include  tr^nagjnQ;  invsr4ww*.  financial 

psduetflrux  pfciecladeehcn, 
etc 


A  CtflllTs 


Objective  |  Method  |&™tranti  Deo=»in  Variables 

ICJ  ITh» 


R]  Optimization  Complete 


Optimization  Result 


Redacted 


Problem  Parameters: 
Number  of  variables  is 

23 

Number  of  functions  is 

2 

Objective  function  will 

be  MAXimized 

Starting  \ 

Functions: 

Function 

Initi 

No.  Name 

Status 

Type 

vali 

1  G 

OBJ 

2848. 1C 

2  G 

variabl  es: 

RNGE 

495 . OC 

variable 

initi 

no.  Name 

Status 

vali 

Optimal  values  have  been  found.  Do  you  wish  to  replace 
revert  to  the  original  inputs? 


Replace 


H* 

Bray 

H5 

Bray 

HG 

Bray 

H7 

Bray 

HS 

bury 

H9 

pray 

HID 

Stay 

L_H  i 

Rj  Optimization  Su 


ary 


Optimization  is  used  to  allocate  resources  where  the 
results  provide  the  max  returns  or  the  min  cost'risks. 
Uses  include  managing  inventories,  financial 
portfolio  allocation,  product  mix,  project  selection, 
etc. 


Optimization 


.  Objective 


Method 


|  Constraints  |  Statistics  |  Decision  Variables 


#  Static  Optimization 

Run  on  static  model  without  simulations.  Usually  run  to  determine  the 
intial  optimal  portfolio  before  more  advanced  optimizations  are  applied. 
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Dynamic  Optimization 

Asimulation  is  first  run,  the  results  of  the  simulation  are  applied  in  the 
model,  and  then  an  optimization  is  applied  to  the  simulated  values. 


N  u  mber  of  Si  niul  ation  T rials 


I  iT^] 


Stochastic  Optimization 

Similar  to  dynamic  optimization  but  the  process  is  repeated  several 
'  decision  variables  will  each  have  its  own  forecast 
its  optimal  range. 

r 


I9W 

I&30 

1620 

1750 

lOU'J 


Efficient  Frontier 


?Cin 


7^0 


Budget 

rra-Vn"'1 


Copy 


iulation  Trials 
imization  Runs  f" 


ICOO^j 


L 


OK 


j  Cancel 


An  Efficient  Frontier 
Analysis  provides 
optimal  selection  for  a 
variety  of  budget 
possibilities 
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Risk  simulation  of  cost  provides  the  decision  maker  with  additional  data 


Risk  analysis  and 

100,000 

simulation  trials 
on  cost 
estimations... 

We  can 
determine  the 
probability  that 
ACB-X  will 
exceed  the  $150M 
budget, 

determine  what 
$171 M  will  yield  a 
99%  certainty  of 
sufficient  budget 
to  cover  all  costs 

We  also  looked  at 
the  optimal 
portfolio  given  a 
90%  probability 
that  $150M  will  be 
enough 


*11  n  p> 


Total  Cost  -  Risk  Simulator  Forecast 


0  E3 


Histogram  Statistics  Preferences  Options  Controls 


~r i  r- nnnn  t^Ifi 


Type  |  Right-Tail  £  ^  |  150.0000  Infinrry  Certainty  %  |  03.30-^ 


R|  Total  Cost  -  Risk  Simulator  Forecast 


Histogram  Statistics  Preferences  Options  Controls 


[°1 


Type  |  Left-Tail  = 

-Infinity 

150.0000 

Certainty  %[  16.70-jH 

R;  Total  Cost  -  Risk  Simulator  Forecast 


[Hi 


Histogram 


Statistics 


Preferences  Options 


Controls 


Statistics 


Result  | 


Number  of  Trials 


Median 

Standard  Deviation 
Variance 

Coefficient  of  Variation 

Maximum 

Minimum 

Range 

Skewness 

Kurtosis 

25";  Percentile 

75";  Percentile 

Percentage  Error  Precision  at  05=4  Confidence 


1000 

33 

155.S004 
6.4400 
41 .5091 
0.0412 
174.9407 
142.0720 
31 .9769 
0.3655 
-D.4G4S 
151.6747 
160.0706 
0.2553"; 


*Screen  shots  from  Risk  Simulator  software 
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Optimized  portfolios  are  time-sequenced  and  risks  are  quantified 
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Probability  of  under  Budget  ACB14 

Pmtiannty  or  under  Budget  agbig 

Probability  of  Under  Budget  ACBtS 


ACB16  ACB  16 


ACBI  6 


AC  B  IB 


ACB  f4  Median  SQtn 
ACB  14  Medan  85m 
ACBi  4  Median  95m 
ACB  1 6  Median  50tn 
ACB  16  Median  85m 
ACB  16  Median  95m 
ACBi 3  Median  50m 
ACB  id  Median  85tn 
ACBi 8  Mediae  95m 


Percentile 

Percentile 

Percentile 

Percentile 

Percentile 

Percentile 

Percentile 
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on  Budget 
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on  Budget 
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Optimal  Optimal  i  J 

on  Budget  Cost-Risk  Mu$t-Hav£  Cost-Risk 
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2 

ft 

7 

9 

10 

1 

2 

4 

7 

1  3IQ-93 

299.74 

115,5ft 

61,02 

1  I43rB7 

151,58 

2ftftr03 

230,96 

42.24 

S7.94 

127-93 

151.58 

j  $1.46.00 

$139-00 

3149.00 

$129-09 

$141.00 

S129-00 

$150-00 

S137-W 

sufioo 

$35-06 

314J.O0 

5179-00 

5413.00 

$363.00 

$440.00 

S39S.OO 

29.710% 

97-90% 

41.SO% 

90.eo% 

72.23% 

90.90% 

16.25% 

99.90% 

\  94.30H 

99.90% 

73 M% 

90,90% 

1  £  153. 20 

$142.90 

£192.60 

$132.30 

£160.22 

$146,60 

£166.60 

$146,30 

£164.30 

$143.70 

£173.90 

$159,50 

j  $145.40 

$137,90 

£196.50 

$139,50 

£153.50 

$147,30 

£164.60 

$143,30 

1  £197.60 

$152.70 

£169.20 

$145,30 

$126.90 

$95.30 

$145.10 

$137.90 

j  $142.90 

$101.30 

£193.50 

$147.30 

|  £190.20 

$104.30 

£196.40 

$152,70 
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More  complex  options  and  constraints  can  be  accommodated  in  the  analyses 
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Inclusion  of  Analysis  of 
Alternatives  or  Courses 
of  Action  using 
strategic  real  options 
analysis  of  various 
implementation 
pathways ,  suitable  for 
nested  and  path 
dependent  options 
(some  components  and 
capabilities  are 
interdependent) 


Strategy  Tree 

(Complex  Multi-Stage  Development) 


In  reality,  an  R&D  project  will  yield  intellectual  property  and  patent 
rights  that  the  firm  can  easily  license  off  (Abandon).  In  addition,  at  any 
phase,  the  project’s  development  can  be  slowed  down  (Contract)  or 
accelerated  (Expand)  depending  on  the  outcome  of  each  phase. 


rt 


Phase  IV 
Inv  =  S20M 


HInv=  $20Mto  spin  off 
into  new  pro  duct  yielding 
50%  higher  market  share 


Reduce  R&D  spending  by 
$20M  and  pro  fits  by  50% 


Spread  out  R&D  investments 
overtime.  Spend  a  little  over 
time  to  decide  if  this  new 
emerging  technology  is  viable. 
The  firm  can  cut  its  losses  and 


Abandon  and  sell  off  assets  or 
intellectual  property  for  $15 


Risk  can  be  mitigated  or  planned  for  though 

-  Budgeting  to  the  amount  that  the  simulator  reveals  is 
necessary  to  provide  a  given  level  of  risk 

-  Improving  cost  estimate  quality  and  reducing  volatility 

-  Up-front  action  to  change  the  cost  equation 

•  Contractual  limits  on  cost  through  use  of  fixed  price  or 
other  contract  vehicles  to  shift  risk  with  the  vendor 

•  Incentives  to  the  contractor  that  reward  for  success  and 
penalize  for  failure  to  meet  cost  targets 

Capability  selection  can  also  be  accomplished  by  applying  risk 
constraints  during  the  optimization 
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Going  forward 

•  More  complex  analyses  to  determine  which  optimization 
portfolio  to  choose 

-  nested  and  mutually  exclusive  options  among  and  between  various 
capabilities 

-  expansion  options  of  a  base  capability  into  additional  capabilities 

•  Strategic  real  options  approach  to  generate  different 
implementation  pathways 

-  provide  strategic  option  trees 

-  identify  best  decision  strategic  option  pathway 


Back  up  slides 


Risk  Simulation  (Risk  Management)  shows  the  range  of  likely  outcomes 


^R]  Return  on  Investment  -  Risk  Simulator  Forecast 
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Histogram  Statistic  Preferences  Options 


Return  on  Investment  ft 000  Trials} 


Type  | Two-Tail  3  21222.569* 


Certainty  %  |  OD.OQ^j 


Q  Return  on  investment  -  Risk  Simulator  Forecast 


I .  C3  |  ©  |— 


Histogram  Statistic  |  Preferences  |  Options. 


D  of  i  i  r-rfc  I  r.i  rc.rf  n-;c.r.f  1  AHA  T  pia  I  r  'i 


21625  22525  23525 


Type  |  Right -Tail  2ZOOO.OOW  | 


Certainty  X  £2.B0*^i 


ProfitKScore  -  Risk  Simulator  Forecast 
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Histogram  Statistic  Preferences  Options 
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Profit* Score  (fOOp  Trials) 
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1.1 

1.0 

D9 

0.3 

0.7 

0.5 

D.5 

0.4 

9.3 

02 
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Type  |  Two-Tail  T  | 


Certainty  %  |  50.00-1-1 


!r]  Profit*Score  -  Risk  Simulator  Forecast 


Histogram  Statistic  Preferences  Options 


Statistics 

Result  \ 

Number  of  Trials 

1000 

Mean 

1 .24401  5E-fQ0G 

Median 

1 .24441 5E+O06 

Standard  Deviation 

3-.342542E-i-DC'4 

Variance 

1.117259E+O09 

Coefficient  of  Variation 

0.0269 

Maximum 

1 .35B625E+O06 

Minimum 

1.149545E+006 

Range 

2.090753E+005 

Skewness 

0.0500 

Kurtosis 

T>.2142 

25%  Percentile 

1.22163QE+006 

75%  Percentile 

1 .2&5566E+O06 

Percentage  Error  Precision  at  55%  Confidence 

0.1664% 

I  c=»  ©  m 
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Risk  Analytics  (sensitivity,  tornado,  fitting,  and  many  other 
analytical  tools)  provide  depth  of  understanding 


Tornado  Chart 


I  Distribution  Fitting  Result 


1  Distribution 

Test  Statistics 

|  P-Value 

|  Rank 

1  Normal 

1  0.02 

1  99.96  V, 

!  1  ■  1 

Gamma 

0.03 

93.33  V 

2 

Lognonnal 

0.03 

93.37  V 

3 

Logistic 

0.03 

97.19  V 

4 

Gumbel  Minimum 

0.05 

73.91  V 

5 

Gumbel  Maximum 

0.05 

57.47  V 

6 

Cauchy 

0.07 

26.53  V 

7 

Triangular 

0.03 

15.90  V 

3 

Chi-Square 

0.10 

3.11  V 

9 

Pareto 

0.15 

0.04  V 

10 

T 

0.36 

0.00  V 

11 

Exponential 

0.42 

0.00  V 

12 

F 

0.92 

0.00  V 

13 

Weibull 

1.00 

0.00  V 

14 

Uniform 

1.00 

0.00  V 

15 

Rayleigh 

1.00 

0.00  V 

16 

Beta 

1.00 

0.00  V 

17 

Statistical  Summary 


Normal 

Mean  =  100.67 
Sigma  =  10.40 

Kol  mogorov-Smi  rno\ 
Test  Statistic:  0.02 
P- Value:  00. OS  V 


Mean 

Stdev 

Skewness 

Kurtosis 
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Nonlinear  Rank  Correlation  (Net  Present  Value) 


H  0.64,  Product  B 
|-0.50,  Effective  Tax  Rate 
0.48,  Product  A  Sale  Quantity  ('0 

0.21 ,  Price  Erosion  Rate 
0.20,  Product  A  Avg  PriceflJnit 
0.20,  Product  C  Avg  PriceAJnit 
0.1 8,  Product  C  Sale  Quantity  ('000s) 

■  0.1 2,  Product  B  Avg  PriceflJnit 
B0.03,  Annualized  Sales  Growth  Rate 


0.1 


0.2 


0.3 


0.4 


0.5 


0.6 


0.7 


0. 055  0.045 

0.01 8  P  0.022 

9  I  11 


Percent  Variation  Explained  (Net  Present  Value) 


1 41 .60%,  ProdL 


]  24.53%,  Effective  Tax  Rate 
1 23.1 7%,  Product  A  Sale  Quantity  ('000s) 


4.49%,  Price  Erosion  Rate 

k  Avg  PriceflJnit 


:  Avg  PriceflJnit 
Sale  Quantity  ('000s) 
g  PriceflJnit 
;s  Growth  Rate 


0.2  0.3 


0.3  0.4 


0.4  0.5 
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